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FOREWORD 


The  Army  Research  Institute  for  the  Behavioral  and  Social 
Sciences  (ARI)  conducts  research  on  tactical  information  systems 
with  particular  emphasis  on  the  human  factor  in  battlefield 
command/control  and  intelligence  functions  and  operations.  The 
support  and  extension  of  human  capabilities  by  computer  technol- 
ogy is  necessary  to  meet  the  challenge  of  the  modern  battlefield. 
Cost  considerations  call  for  careful  attention  to  the  performance 
increment  attainable  for  alternative  computer  applications  and 
aids.  The  ARI  research  program  in  this  domain  is  independently 
and  jointly  executed  by  the  Battlefield  Information  Systems 
Technical  Area  in  Alexandria,  VA  and  the  ARI  Field  Unit  at  Fort 
Leavenworth,  Kansas. 

The  present  report  describes  research  accomplished  by  the 
Fort  Leavenworth  Field  Unit  to  determine  the  efficacy  of  a 
Functional  Applications  Package  (FAP)  designed  to  automate  order 
of  battle  and  target  intelligence  files.  The  design  and  evalu- 
ation of  candidate  FAP's  for  inclusion  in  the  requirement  speci- 
fications for  the  developing  Tactical  Operations  System  (TOS)  is 
part  of  the  Army's  overall  effort  to  insure  a user  oriented, 
performance  enhancing,  affordable  TOS. 

This  research  was  done  under  Army  Project  2Q763743A774, 
"Man/Machine  Interface  in  Integrated  Battlefield  Control  Systems" 
in  response  to  special  requirements  of  the  Combined  Arms  Combat 
Development  Activity,  Fort  Leavenworth,  Kansas. 

IfllEPH  ZEIDNER 
Technical  Director 


UTILITY  OF  AUTOMATION  OF  ORDER  OF  BATTLE  AND  TARGET  INTELLIGENCE 
DATA  FOR  INTELLIGENCE  ANALYSIS 

BRIEF 


Requirement : 

To  evaluate  Order  of  Battle  (OB)  and  Target  Intelligence 
(TI)  files  as  additions  to  the  Enemy  Situation  (ENSIT)  data  base 
of  the  Tactical  Operations  System  Test  Bed  Simulation,  with 
special  emphasis  on  the  effect  of  automating  the  files  on  the 
speed,  accuracy,  and  completeness  of  performance. 


Procedure : 

As  part  of  an  on-going  research  program  to  develop  software 
applications  for  inclusion  in  the  Army's  Tactical  Operations 
System  (TOS),  a series  of  Functional  Application  Packages  (FAPs) 
is  being  developed  at  Fort  Leavenworth,  Kansas.  FAPs  that  are 
favorably  evaluated  will  be  included  as  requirements  for  the 
prototype  TOS  Definition.  Evaluation  of  the  first  of  the  series 
is  the  subject  of  the  present  report.  Twenty  qualified  intelli- 
gence analysts  participated.  Ten  analysts  attempted  to  accom- 
plish a series  of  intelligence  tasks  with  the  aid  of  the  simu- 
lated basic  TOS,  including  automated  OB  and  TI  files  (experimen- 
tal condition);  the  remaining  10  attempted  to  accomplish  the  same 
tasks  with  the  aid  of  the  basic  simulated  TOS  and  manual  OB  and 
TI  files  (control  condition.  Performance  was  evaluated  in  terms 
of  timeliness  (task  completion  time),  accuracy,  and  completeness. 


Findings : 

Automating  OB  and  TI  files  significantly  increased  the  accu- 
racy and  completeness  with  which  selected  intelligence  require- 
ments were  processed.  However,  task  completion  time  did  not 
differ  significantly  for  the  two  groups.  Information  could  be 
retrieved  from  manual  files  as  quickly  as  from  automated  files, 
through  the  use  of  a data  base  management  system. 


Utilization  of  Findings: 

The  findings  of  the  investigation,  in  conjunction  with 
other  cost-effectiveness  considerations,  provide  basic  data  for 
use  in  deciding  whether  or  not  the  FAP  should  be  included  in  the 
Required  Operations  Capabilities  document  for  TOS. 
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UTILITY  OF  AUTOMATION  OF  ORDER  OF  BATTLE  AND  TARGET  INTELLIGENCE  DATA 
FOR  INTELLIGENCE  ANALYSIS 


BACKGROUND 

The  need  to  develop  a capability  to  store,  retrieve,  correlate,  and 
filter  increasing  amounts  of  information  for  input  by  battlefield  systems 
to  a division  has  led  to  the  establishment  of  an  Army  work  program  to 
determine  user  requirements  for  a Tactical  Operations  System  (TOS).  The 
U.S.  Army  Combined  Arms  Combat  Developments  Activity  (USACACDA)  is  the 
Army's  proponent  for  TOS;  the  Combined  Arms  Center  (CAC) — specifically, 
the  Command  Systems  (CS)  Division  of  USACACDA — is  in  the  process  of 
defining  user  requirements  for  a TOS.  The  FY  76  work  of  the  CS  Division 
concentrated  on  developing  requirements  for  a division  level  TOS  that 
was  intended,  initially  at  least,  to  support  the  command  and  control 
process  by  increasing  the  ability  of  the  staff  (primarily  the  G2)  to  make 
more  timely  and  accurate  intelligence  (as  well  as  combat  information) 
available  for  the  commander's  consideration. 

An  automated  system  to  support  command  and  control  activities  at 
higher  levels  of  tactical  command  is  needed  because  of  the  development 
of  other  automated  data  acquisition  and  processing  systems,  e.g.,  the 
Tactical  Fire  Direction  System  (TACFIRE),  the  Position  Location  and 
Recording  System  (PLARS),  and  the  Remotely  Monitored  Battlefield  Sensor 
System  (REMBASS).  The  proliferation  of  these  automated  information  and 
collection  and  processing  systems  underscores  the  need  for  development 
of  a command  and  control  ("centroid")  system  that  has  the  capability  of 
storing,  retrieving,  correlating,  and  filtering  the  diverse  and  large 
volumes  of  information  that  will  flow  from  other  specialized  acquisition 
and  processing  systems  into  division  level  command  posts. 

CAC  has  established  a review  panel  to  monitor  and  steer  the  TOS 
requirements  definition  program.  As  applications  are  identified,  the 
Review  Panel,  which  includes  Army-wide  representation,  reviews  candidate 
applications  and  makes  a decision  as  to  whether  or  not  they  should  be 
developed  and  evaluated  (validated). 

Each  software  application1  being  developed  for  TOS  is  referred  to  as 
a Functional  Applications  Package  (FAP).  Also,  each  FAP  is  designed  to 
support  or  build  on  its  predecessor.  The  Army  Research  Institute  (ARI) 
has  an  on-going  program  of  research  at  Fort  Leavenworth  to  identify  and 
evaluate  potential  automation  assists  for  the  commander  and  staff. 


1 Note  that  each  potential  application  CAC  identifies  is  programmed  to 
run  on  commercial  hardware,  primarily  the  CDC  6500/6400  computer  in 
the  Data  Processing  Field  Office  of  the  Army  Training  and  Doctrine 
Command  (TRADOC).  Thus,  CAC  simulates  each  potential  requirement 
it  identifies  for  the  eventual  TOS  system  and  evaluates  that  appli- 
cation to  determine  if  it  is  indeed  a valid  requirement  which  should 
be  included  in  the  prototype  requirements  specification. 


As  each  FAP  is  identified  and  designed,  it  is  evaluated  by  ARI  to 
determine  if  the  addition  of  the  application  significantly  enhances  the 
performance  of  the  individual  or  group  of  individuals  that  the  appli- 
cation was  intended  to  serve.  The  results  of  the  evaluation  then 
provides  feedback  to  USACACDA  concerning  how  their  own  resources  should 
be  utilized  in  the  future,  for  example,  whether  to  modify  or  continue 
development  of  the  application  or  reject  it  as  a candidate  for  the  TOS 
prototype  definition. 

Results  are  intended  as  preparation  for  the  TOS  Army  Systems  Acquisi- 
tion Review  Council/Department  of  Defense  Systems  Acquisition  Review 
Council  (ASARC/DSARC)  II  meetings.  Results  also  provide  TRADOC's  System 
Analysis  Agency  (TRASANA)  with  data  which  could  be  used  in  the  cost 
effectiveness  analysis  of  TOS,  a requirement  for  the  ASARC/DSARC  II 
meetings. 

On  24  October  1975,  the  CAC  TOS  Review  Panel  met  and  approved  the 
first  Functional  Application  Package  (FAP  1)  for  development  and  evalu- 
ation, as  appropriate.  The  specific  application  package  consisted  of 
the  Order  of  Battle  (OB)  and  Target  Intelligence  (TI)  data  files.  The 
OB  and  TI  files  were  developed  to  "round  out"  the  automated  Enemy 
Situation  (ENSIT)  data  base. 

An  OB  file  was  deemed  necessary  to  serve  as  a repository  for  data 
required  for  detailed  analysis  of  enemy  unit  capabilities  vis-a-vis 
combat  effectiveness,  status  of  men  and  equipment,  and  commander's 
identification  and  personality  characteristics.  The  target  intelligence 
file  was  designed  to  provide  a capability  for  storing  data  on  past 
targets  and  potential  targets.  It  was  designed  (to  the  extent  possible) 
to  be  compatible  with  TACFIRE  message  formats  and  thus  to  facilitate 
TOS's  eventual  interface  with  the  Field  Artillery's  Tactical  Fire 
Direction  System. 


PURPOSE 

ARI  designed  and  conducted  an  experiment  to  evaluate  the  effective- 
ness of  the  first  FAP,  which  was  developed  during  FY  76.  The  experiment 
was  designed  to  determine  how  quickly,  accurately,  and  completely  intel- 
ligence-related tasks  could  be  accomplished  with  the  aid  of  automated 
OB  and  TI  files  augmenting  the  basic  TOS  system  in  comparison  with  the 
same  intelligence  tasks  accomplished  with  the  aid  of  the  basic  TOS 
system  and  manual  OB  and  TI  files.  (See  Appendix  A for  references  which 
describe  the  basic  CAC  simulated  TOS  system,  before  addition  of  FAP  1.) 

A secondary  purpose  of  the  experiment  was  to  identify  strengths  and 
weaknesses  of  the  automated  OB  and  TI  files  in  terms  of  the  number  and 
types  of  data  elements  designed  into  them. 


METHOD 


SUBJECTS 

TWenty  military  intelligence  analysts,  ten  in  each  of  two  treatment 
conditions,  participated  in  the  experiment.  The  analysts  all  had  MOSs 
that  would  qualify  them  to  work  in  the  analysis  and  production  element 
of  the  Division  G2  Section. 


RESEARCH  DESIGN 

A simple  experimental  group  versus  control  group  design  (treatment 
versus  no  treatment)  was  used  to  evaluate  the  effects  of  automating  OB 
and  TI  data  files.  The  two  groups  were  approximately  balanced  in  terms 
of  analysts'  length  of  military  service  and  MOS-related  experience 
(combat  and  peacetime).  Analysts  in  the  experimental  group  had  access 
through  the  General  On-line  Query  System  (GOQS)  to  information  resident 
in  the  ENSIT  data  base,  including  OB  and  TI  data.  In  the  control 
condition,  subjects  had  access  to  all  information,  via  GOQS,  except  that 
in  the  OB  and  TI  files.  They  could  obtain  this  information  from  manual 
card  files  to  which  they  had  access.  Thus,  both  groups  had  access  to 
identical  information,  the  experimental  group  to  automated  files,  the 
control  group  to  manual  files.  The  manual  OB  and  TI  files  used  by  the 
control  subjects  were  constructed  for  that  purpose  by,  OB  analysts  from 
Fort  Hood,  and  thus  were  representative  of  those  currently  used  in  the 
field. 


PROCEDURE 

Analysts  in  both  experimental  and  control  conditions  were  required  to 
answer  a series  of  nine  questions.  The  questions  were  generally  represen- 
tative of  questions  an  OB  analyst  in  the  analysis  and  production  element 
of  the  Division  G2  Section  would  be  expected  to  answer.  The  questions 
used  are  presented  in  the  Technical  Supplement. 

For  each  of  the  first  eight  questions,  the  analysts  in  both  condi- 
tions were  evaluated  in  terms  of  the  time  required  to  answer  the  ques- 
tion, the  accuracy  of  the  response,  and  the  completeness  of  the  response. 

A "correct"  or  best  answer  was  not  available  for  the  ninth  question 
which  required  a summary  estimate  of  enemy  capabilities.  This  question 
was  included  to  obtain  data  as  analysts  constructed  a solution  on  the 
types  of  queries  generated,  the  frequency  of  accessing  the  ENSIT  data 
files  and  the  particular  data  codes  within  files  that  were  accessed. 

This  kind  of  information  can  be  useful  for  evaluating  the  utility  of  the 
system's  data  files.  Due  to  a breakdown  in  computer  system  procedure 
these  data  are  not  addressed  in  this  report.  A detailed  description  of 
the  experimental  procedure  and  variables  that  were  controlled  to  prevent 
their  biasing  the  results  is  provided  in  the  Technical  Supplement. 


FINDINGS  AND  CONCLUSIONS 


The  experimental  group's  performance  was  significantly  superior  to 
the  control  group's  performance  in  terms  of  accuracy  anl  completeness. 
There  were  no  differences  in  the  performance  of  the  two  groups  in  terms 
of  the  speed  with  which  the  eight  tasks  were  accomplished.  Thus,  it  was 
concluded  that  the  addition  of  the  FAP  1 module  to  the  basic  TOS/GOQS 
test  bed  improved  the  quality  of  the  performance  of  those  individuals 
that  the  application  was  primarily  designed  to  support.  Note  that  only 
one  aspect  of  the  total  system  was  evaluated,  and  that  the  evaluation 
took  place  within  highly  controlled  and  structured  conditions.  There- 
fore, caution  should  be  used  in  generalizing  these  results  to  any  other 
aspect  of  the  system  or  to  the  system's  overall  usefulness  in  a field 
environment . 


DISCUSSION 

The  results  of  this  experiment  support  two  of  the  major  suppositions 
that  underly  the  development  of  TOS,  as  well  as  command  and  control 
support  systems  in  general.  Both  the  reliability  and  completeness  of 
reports  and  data  extracted  or  transmitted  by  the  automated  system  are 
greater  than  can  be  achieved  through  use  of  manual  procedures.  These 
findings,  of  course,  were  obtained  with  a "clean"  data  base — one  that 
contained  few,  if  any,  errors,  and  was  obtained  under  highly  structured 
or  controlled  conditions.  The  results  did  not  support  the  third  supposi- 
tion, perhaps  the  most  significant  one,  that  retrieval  of  information 
through  the  use  of  automation  is  faster  than  manual  processing.  This 
finding  could  very  well  be  an  artifact  of  the  size  of  the  data  base 
used,  which  consisted  of  approximately  200  messages,  a number  much 
smaller  than  one  would  expect  to  develop  in  a combat  environment  where 
messages  are  received  from  many  sources,  both  internally  (within  the 
division  TOC  or  command  post)  and  externally  (from  subordinate  and 
attached  units  and  perhaps,  eventually,  from  other  divisions,  from 
corps,  and  from  other  tactical  data  systems).  As  the  size  of  a data 
base  or  the  volume  of  message  traffic  increases,  the  benefits  of  automa- 
tion, in  terms  of  the  speed  with  which  this  volume  of  message  flow  can 
be  processed,  accessed,  and  transmitted,  should  increase  appreciably. 

This  conjecture,  obviously,  assumes  that  expedient  methods  (man/machine 
systems)  are  developed  for  entering  data  into  the  system. 

The  findings  of  this  analysis  may  also  be  interpreted  in  terms  of 
the  criteria  that  may  be  used  in  combat  situations  or  simulated  combat 
situations  to  evaluate  the  performance  of  intelligence  personnel  like 
those  who  participated  in  the  present  study.  On  a prima  facie  basis  and 
especially  considering  the  size  of  the  data  base  used  in  this  experiment, 
the  findings  are  rather  surprising.  If  anything,  one  would  have  expected 
no  performance  differences  for  the  accuracy  and  completeness  criteria, 
since  the  same  information  was  available  to  the  subjects  in  both  condi- 
tions and  in  a very  concise  file  form  for  the  subjects  in  the  manual 
(control)  condition.  Also,  the  subjects  in  the  control  condition 
were  told  to  work  quickly,  yet  not  to  allow  accuracy  and  completeness 
to  suffer.  Thus,  they  were  not  really  under  time  pressure  per  se. 
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However,  the  control  subjects  did  complete  the  tasks  given  them  relatively 
quickly,  and  the  accuracy  and  completeness  of  their  responses  suffered. 

In  combat  situations  and  simulated  combat  situations,  it  is  probably 
often  impossible  to  determine  precisely  whether  reports  received  or 
intelligence  tasks  accomplished  are  totally  reliable.  It  is  extremely 
easy  in  these  situations,  however,  to  determine  whether  or  not  a given 
task  is  accomplished  quickly.  Thus,  performance  in  these  situations  is 
probably  evaluated,  for  the  most  part,  in  terms  of  its  timeliness,  which 
results  in  strengthening  this  performance  dimension  to  the  detriment  of 
other  highly  relevant  performance  dimensions.  If  this  interpretation  is 
correct,  it  suggests  that  the  training  intelligence  personnel  receive, 
particularly  on-the-job  training,  should  stress,  to  the  extent  possible, 
the  tradeoffs  between  expeditious  functioning  per  se  and  expeditious 
functioning  which  includes  rigor  regarding  the  reliability  of  the 
information  processed  and  the  estimates  developed.  This  reasoning 
would  apply  to  the  training  of  G2  Section  staff  to  process  information 
either  manually  or  with  the  assistance  of  automation. 
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TECHNICAL  SUPPLEMENT 


METHOD 


SUBJECTS 

Twenty  intelligence  analysts  participated  in  the  experiment,  ten  in 
each  of  two  treatment  conditions.  All  had  MOSs  that  would  qualify  them 
to  work  in  the  analysis  and  production  element  of  the  Division  G2 
Section. 

The  sample  was  drawn  from  several  locations  or  units,  including 
USAICS,  III  Corps,  the  Fourth  Infantry  Division  and  the  82d  and  101st 
Airborne  Divisions.  An  attempt  was  made  to  equate  the  two  groups  in 
terms  of  length  of  military  service,  MOS-related  experience  during 
combat,  and  MOS-related  experience  during  peacetime,  a set  of  variables 
that  were  conjectured  to  have  potential  impact  on  the  subjects'  perfor- 
mance of  the  tasks  imposed  during  the  experiment2  (Table  1).  One-way 
analyses  of  variance  were  used  to  evaluate  the  similarity/dissimilarity 
of  the  two  treatment  groups  in  terms  of  the  three  variables.  None  of 
these  tests  proved  significant  at  the  .05  level:  (length  of  military 
service  F = .003,  df  = 1,18;  experience-combat  F = .01,  df  = 1,18;  MOS 
experience-peacetime  F = .01,  df  = 1,18),  indicating  that  the  balancing 
of  these  variables  between  the  two  groups  had  been  accomplished  with 
reasonable  success. 


I 


Table  1 

'AVERAGE  MONTHS  OF  MILITARY  SERVICE,  MOS  SERVICE  IN  COMBAT,  AND 
MOS  SERVICE  IN  PEACETIME  FOR  EXPERIMENTAL  AND  CONTROL  GROUPS 


Experimental 

(n-10) 

Control 

(n-10) 

Military 

Service 

MOS  Service 
(Combat) 

MOS  Service 
(Peacetime) 

Military 

Service 

MOS  Service 
(Combat) 

MOS  Service 
(Peacetime) 

X-85.4 

X-4.8 

X-19.5 

X-87.5 

X«4.3 

X=20. 8 

sd-86.55 

sd-10.12 

sd=19.88 

sd-97.14 

sd=8. 06 

sd-29.11 

2 MOS  experience  and  the  time  in  service  of  the  analysts  was  extremely 
variable,  as  reflected  in  Table  1.  For  military  service,  MOS-related 
service  (combat)  and  MOS-related  service  (peacetime),  the  ranges  were 
9-276  months,  0-24  months,  and  2-96  months,  respectively. 


RESEARCH  DESIGN 


The  analysts  performed  a series  of  tasks  in  one  or  the  other  of  two 
treatment  conditions: 

1.  In  the  experimental  condition,  the  subject  was  allowed  to  access, 
through  a computer  terminal  operator  familiar  with  GOQS  and  the  ENSIT 
data  base,  the  information  resident  in  that  data  base  (including  OB  and 
TI  file  data),  to  obtain  answers  for  a series  of  intelligence  related 
questions. 

2.  In  the  control  condition,  the  subjects  were  allowed  to  access 
Information  from  the  ENSIT  data  base  (through  the  same  terminal  operator 
available  for  the  experimental  subjects)  excluding  the  OB  and  TI  files. 

OB  and  TI  data  were  available  to  these  subjects  only  in  manual  file 
form.  Hie  manual  OB  and  TI  files  contained  information  identical  to 
that  in  the  automated  files. 


DEVELOPMENT  OF  THE  MANUAL  OB  AND  TI  FILES 

Two  OB  analysts  (both  holding  MOS  96B40)  were  brought  to  CAC  from  III 
Corps,  Fort  Hood,  Texas,  for  the  purpose  of  developing  the  manual  data 
files.  These  particular  analysts  were  selected  because  of  their  reported 
high  level  of  expertise  in  the  OB  area.  A printout  of  the  data  base  that 
had  been  generated  to  support  a demonstration  of  FAP  1 and  the  present 
experiment  (based  upon  a Command  and  General  Staff  College  scenario, 

R1220 — a non-classified  scenario  developed  from  SCORES  Europe  1,  sequence 
2A)  was  provided  to  the  analysts.  After  being  Instructed  how  to  extract 
appropriate  data  from  the  printout,  the  analysts  were  told  to  develop  a 
manual  file  of  the  information,  in  a form  which  they  used  in  division 
CPX's  (and,  in  the  case  of  one  analyst,  as  he  had  used  during  combat  in 
Vietnam) . 

After  the  analysts  had  finished  the  manual  file,  two  project  officers 
(a  major  and  a lieutenant  colonel)  from  the  Command  Systems  Division 
Independently  checked  the  contents  of  the  manual  file  against  the 
computer  printouts  of  the  OB  and  TI  files  to  verify  that  all  information 
from  the  printouts  had  been  accurately  and  completely  transferred. 

Minor  editings  were  made  on  the  basis  of  this  review. 


SUPPORT  PERSONNEL 

Two  support  personnel,  a computer  terminal  operator  and  the  experi- 
menter, were  present  when  each  participant  was  run.  The  computer 
terminal  operator  retrieved  data  from  the  computer's  base  for  the 
participant.  In  essence,  he  represented  the  analyst's  training  in  the 
use  of  the  computer  system  and  thus  served  as  a "servo"  for  the  subject. 
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The  computer  terminal  operator  was  a member  of  BDM  Services  Company's 
staff,  the  CS  Division's  contractor  who  developed  FAP  1 as  well  as  the 
other  ENSIT  files  and  GOQS.  This  operator  was  also  responsible  for 
coding  and  loading  the  data  base  that  supported  the  experiment.  Thus, 
he  was  thoroughly  trained  in  translating  natural  language  questions  into 
correct  system  compatible  queries  and  could  translate  or  interpret  the 
resulting  output  for  the  subject. 

The  experimenter  was  present  to  answer  any  questions  the  analyst 
might  have  regarding  the  general  purpose  of  the  experiment,  the  role  he 
was  to  play,  and  the  task(s)  he  was  requested  to  perform.  The  experi- 
menter was  allowed  to  answer  only  general  questions  regarding  the 
specific  task.  He  could  explain  what  information  the  question  asked 
for,  but  was  not  permitted  to  explain  or  suggest  what  questions  the 
analyst  might  need  to  ask  in  order  to  satisfy  the  requirement.  The 
experimenter  also  recorded,  with  a stopwatch,  the  following  time 
intervals : 

1.  Task  completion  time.  Elapsed  time  between  presentation  of  a 
task  to  an  analyst  and  the  analysts'  completion  of  that  task.  Task 
completion  time  was  recorded  for  each  subject  on  the  form  attached  as 
Appendix  B. 

2.  System  query  composition  time.  Time  required  for  the  computer 
terminal  operator  to  construct  and  "send"  a system  compatible  query 
after  receiving  an  information  request  from  the  analyst  (aborted  attempts, 
i.e.,  construction  of  a system  incompatible  query  and  its  rejection  by 
the  system,  were  also  recorded3).  Although  the  computer  terminal 
operator  was  proficient  with  the  system  at  the  outset  of  the  experiment, 
it  was  necessary  to  gather  data  on  his  performance  during  all  trials  for 
both  treatment  conditions  in  order  to  verify  that  performance  improvements 
(if  any)  were  equated  for  the  two  groups.  Composition  time  was  recorded 
on  the  form  attached  as  Appendix  C. 

3.  Output  annotation  time.  The  interval  of  time  between  the  oper- 
ator 's  receiving  hard  copy  from  the  computer  and  translating  (annotating) 
that  output  in  a form  understandable  to  the  analyst.  Annotation  time  was 
also  recorded  on  the  form  attached  as  Appendix  C. 


Note  that  the  terminal  operator  might  have  to  compose  more  than 
one  system  query  to  provide  the  information  requested  by  the  analyst. 
Operator  query  composition  time  was  recorded  for  each  sub-question, 
whether  correct  (system  compatible)  or  incorrect  (system  incompatible). 
Also,  whether  or  not  the  query  the  operator  constructed  met  the 
requirements  stated  by  the  analyst  was  recorded , as  it  was  possible 
the  operator  could  have  misunderstood  what  the  analyst  requested. 
Appendix  C contains  the  form  that  was  used  to  document  terminal 
operator  performance  for  those  dimensions  discussed. 
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4.  Computer  response  time . The  computer  that  was  used  in  the  experi- 
ment is  shared  on  a time  basis  with  a large  number  of  other  users. 

Thus,  turnaround  time  was  dependent  upon  the  capacity  of  the  machine  and 
the  number  of  users  it  was  serving  at  any  given  poirtt  in  time.  Because 
the  machine  could  not  be  relied  upon  to  provide  a consistent  response 
Interval,  its  performance  was  also  recorded.  Specifically,  two  time 
intervals  were  recorded:  (1)  Post  query  computer  response  time,  the 
interval  of  time  from  "sending"  (command  to  execute  a query)  to  receipt 
of  a response  in  hard  copy  form  (either  an  error  message  or  the  data 
requested);  and  (2)  "Call  G"  response  time.  In  order  to  input  a GOQS 
query,  it  is  first  necessary  to  initiate  a "Call  G”  command.  This 
action  essentially  commands  the  system  to  prepare  itself  for  the  entry 
and  subsequent  execution  of  a GOQS  query.  Again,  the  time  necessary  to 
process  this  command  was  dependent  upon  the  state  of  the  computer 
system,  i.e.,  the  number  of  other  jobs  being  processed  at  a given 
moment.  The  lag  time  between  the  "Call  G"  command  and  the  receipt  of  a 
"clear"  response  from  the  computer  along  with  post  query  computer 
response  time  was  recorded  on  the  form  shown  in  Appendix  C for  each 
transaction  within  trials  for  both  treatment  groups. 


PROCEDURE 

Prior  to  actual  participation  in  the  experiment,  the  experimenter 
provided  each  analyst  with  a set  of  general  instructions  that  specified 
the  purpose  of  the  experiment  and  in  what  capacity  the  analyst  was  to 
support  the  effort.  These  instructions  are  included  as  Appendix  D,  Part 
I and  Part  II,  for  the  experimental  and  control  conditions,  respectively. 
After  the  analyst  read  these  instructions,  he  was  given  the  following 
material  to  study: 

1.  A composite  map  made  up  of  the  following  sections:  USACGSC 
50-229,  Europe,  Sheet  1,  (Friedburg  - Frankfurt),  Edition  1974,  1:50,000; 
USACGSC  50-245,  Europe,  Sheet  1,  (Fulda  - Lauterbach) , Edition  1975, 
1:50,000;  USACGSC  50-246,  Europe,  Sheet  1 (Schluchtern  - Bad  Kissingen), 
Edition  1975,  1:50,000;  USACGSC  50-249,  Germany,  Sheet  1,  (Nidda  - 
Ulmbach) , Edition  1975,  1:50,000;  USACGSC  240-138,  Western  Europe, 

Sheet  1,  (Kassel  - Fulda),  Edition  1975,  1:50,000.  On  this  map  was  an 
overlay  depicting  the  position  of  friendly  units  and  enemy  units  known 

to  be  in  contact  along  the  forward  edge  of  the  battle  area  (FEBA). 

2.  A description  of  the  general  and  special  situation  of  the  54th 
Mechanized  Infantry  Division,  a unit  assigned  to  the  10th  US  Corps  in 
the  R1220  scenario  and  the  one  to  which  the  participant  assumed  he  had 
been  assigned  for  purposes  of  the  experiment.  The  description  of  the 
general  and  special  situation  of  the  54th  Mechanized  Infantry  Division 
is  attached  as  Appendix  E. 

The  analysts  in  both  treatment  conditions  were  given  15-30  minutes 
to  study  the  above  materials,  including  the  general  instructions.  The 
analysts  in  the  control  condition  were  given  an  additional  30  minutes 
to  study  the  composition  and  contents  of  the  manual  OB  and  TI  files 
they  were  to  use  in  the  experiment.  To  assist  them  in  becoming  familiar 
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with  and  retrieving  information  from. these  files,  they  were  requested  to 
provide  answers  for  a selected  set  of  questions.  These  questions  are 
Included  as  Appendix  F.  The  sequence  of  events  subsequent  to  the 
general  orientation  period  are  described  below  separately  for  the 
experimental  and  control  conditions. 


CONTROL  CONDITION 

The  situation  map,  the  description  of  the  general  and  special 
situation  of  the  54th  Mechanized  Infantry  Division,  and  the  manual  OB 
and  TI  files  were  made  available  for  reference  and  use  by  the  analysts 
in  this  condition  throughout  the  remainder  of  the  experiment.  Other 
materials,  including  blank  index  cards  (which  could  be  used  to  update 
the  manual  OB  and  TI  files  as  the  experiment  progressed),  pencils  (both 
lead  and  grease),  and  paper  were  available  as  well.  Only  the  computer 
terminal  operator  and  experimenter  were  present  when  the  analyst  was 
run.  The  experimenter  provided  the  analyst  with  a set  of  detailed 
instructions  at  the  outset  of  this  period.  Basically,  these  instructions 
(Appendix  G)  informed  the  analyst  that  he  was  to  provide  answers  to  a 
series  of  nine  questions  that  would  be  presented  to  him  one  at  a time  and 
that  he  could  access  information  from  his  manual  files,  the  computer's 
data  base,  or  both,  to  satisfy  each  of  the  requirements.  The  questions 
and  the  instructions  as  to  exactly  what  form  his  response  was  to  take 
for  each  question  are  presented  below: 

1.  What  battalion  or  larger  size  units  are  located 
within  Coordinates  NA0087,  NA0077,  NA1077,  NA1087? 

Plot  (using  appropriate  symbology)  all  the  units 
you  have  identified  on  the  situation  map. 

2.  What  are  the  effective  personnel  strengths  of  the  units 
within  Coordinates  NA0087,  NA0077,  NA1077,  NA1087? 

Record  each  unit  and  the  corresponding  strength 
figure  for  it  on  the  data  sheet  provided. 

3.  What  major  items  of  TO&E  equipment  are  operational 
for  the  1st  Battalion  of  the  22d  Motorized  Rifle 
Regiment  of  the  18th  Mechanized  Infantry  Division? 

Record  each  equipment  category  and  the  corresponding 
number  of  operational  items  on  the  data  sheet  provided. 

4.  What  is  the  operational  tube  strength  of  the  1st 
Battalion  of  the  61st  Artillery  Regiment  of  the 
18th  Mechanized  Infantry  Division?  Record  your 
answer  on  the  data  sheet  provided. 

5.  How  many  152  and  122mm  guns  are  located  and  operational 
within  Coordinates  NA0087,  NA0077,  NA1077,  NA1087? 

Record  the  number  of  operational  152  and  122mm  guns 

on  the  data  sheet  provided. 
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6.  What  regimental  and  higher  level  command  posts  are 
located  within  Coordinates  NA0087,  NA0077,  NA1077, 
NA1087?  Plot  (using  appropriate  symbology)  the 

CP  locations  on  the  situatlcyi  map. 

7.  What  bridging  emplacements  on  river  crossing 
points  have  been  identified  within  Coordinates 
NA0087,  NA0077,  NA1077,  NA1087?  Plot  (using 
appropriate  symbology)  these  emplacements  on  the 
situation  map. 

8.  What  FROG/SCUD  sites  are  located  within  Coordinates 
NA0087,  NA0077 , NA1077,  NA1087?  Record  the 
coordinates  of  sites  located  on  the  data  sheet 
provided. 

9.  Will  the  enemy  attack? 

Check  One. 

□ *> 

(a)  If  your  answer  was  no,  stop. 

(b)  If  yor  answer  was  yes: 

L,  Specify  what  course  of  action  will  probably 
be  pursued  by  each  of  the  major  (divisional) 
elements  you  have  identified. 

2.  Where  will  the  breakthrough  occur? 

3.  Who  (what  units)  will  instigate  it? 

4.  What  is  your  best  estimate  of  when  an 
attack  will  occur  (specify  in  hours  and/ 
or  minutes)? 

5.  List  the  authorized  and  effective  personnel 
strengths  for  each  of  the  division  level 
enemy  units  you  think  will  participate  in 
the  attack. 

6.  List  the  major  items  of  equipment  for  each 
of  the  division  level  enemy  units  that  you 
think  will  participate  in  the  attack  and 
indicate  the  effective  strength  of  each 
equipment  category. 


Each  of  the  first  eight  questions  and  the  instructions  that  appeared 
with  them  were  typed  on  5 x 7-inch  index  cards.  The  last  question  was 
printed  on  a separate  sheet  and  appeared  just  as  it  does  above.  The 
questions  were  presented  to  the  participant  one  at  a time  and  each 
succeeding  one  was  presented  only  after  the  analyst  had  signaled  that  he 
had  completed  the  previous  question. 

For  the  first  eight  questions,  the  terminal  operator  knew  where  the 
appropriate  information  resided,  since  this  portion  of  the  experiment 
was  essentially  "canned"  and  had  been  pilot  tested  several  times  in  order 
to  refine  the  procedures  and  to  determine  the  residence  of  the  data 
needed  to  answer  each  question.  Thus,  for  this  set  of  questions,  the 
following  sequence  of  activities  occurred  for  each  question.  The 
experimenter  presented  the  card  on  which  the  question  appeared  to  the 
analyst  and  simultaneously  started  the  stopwatch.  If  the  data  requested 
by  the  analyst  to  answer  the  question  resided  in  his  manual  files,  the 
terminal  operator  immediately  informed  the  analyst  of  this  and  instructed 
him  to  search  for  the  information  there.4  If  the  information  requested 
resided  solely  in  the  computer's  data  base,  the  terminal  operator 
informed  the  analyst  of  this  and  immediately  began  to  construct  a system 
query  that  would  produce  the  information.  Finally,  if  the  data  resided 
in  both  computer  and  manual  files,  the  analyst  was  again  instructed  that 
he  should  begin  retrieving  from  the  manual  files  since  a portion  of  the 
data  did  reside  there.  The  terminal  operator  waited  until  the  analyst 
had  finished  his  search  before  he  began  accessing  the  information  in  the 
computer's  data  base.  (This  procedure  was  sometimes  reversed,  i.e.,  the 
computer  terminal  operator  retrieved  first  and  the  analyst  then  searched 
his  files  for  the  remainder  of  the  information.)  In  the  latter  conditions 
(i.e.,  when  data  were  retrieved  from  the  automated  data  base),  when  hard 
copy  was  received,  the  terminal  operator  deleted  those  portions  of  the 
output  which  were  retrieved  from  the  OB  and  TI  automated  files.  He  then 
annotated  the  remaining  output,  just  as  he  would  have  for  any  other 
system  output.  Following  the  completion  of  any  one  of  these  three 
possible  sequences  for  obtaining  the  requested  information  and  as  soon 
as  the  analyst  had  completed  his  response  in  the  form  specified,  the 
experimenter  recorded  the  elapsed  time  (task  completion  time,  as  defined 
above) . 


4 Note  that  information  in  the  OB  and  TI  files  may  also  be  resident  in 
other  files,  e.g..  Enemy  Unit  Situation  (EUS),  which  were  a part  of 
the  previous  TOS/GOQS  package.  In  Instances  such  as  this  and  for  this 
condition,  the  subject  was  given  the  information  since  it  is  not 
idiosyncratic  to  the  OB  and  TI  files. 
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For  question  9, 5 the  terminal  operator  translated  each  of  the 
analyst's  questions  into  a system  compatible  query,  executed  it,  and 
searched  the  output  to  determine  which  files  were  accessed  to  compose 
the  response.  If  any  of  the  responses  came  exclusively  from  the  OB  and 
TI  files,  that  portion  of  the  output  was  deleted,  and  the  remaining 
output  was  annotated.  The  terminal  operator  then  provided  the  output  to 
the  analyst  and  immediately  informed  him  that  a portion  of  the  data  he 
had  requested  resided  in  his  manual  files.  If  the  system  responded  that 
no  data  were  available  bearing  on  the  participant's  request,  the  operator 
immediately  informed  the  analyst  of  this  and  requested  that  the  analyst 
formulate  another  question.  This  process  was  continued  until  the 
analyst  was  satisfied  that  he  had  obtained  all  the  data  that  he  needed 
to  make  the  estimates  called  for  by  question  9. 


EXPERIMENTAL  CONDITION 

The  procedure  for  participants  in  this  condition  differed  only 
slightly  from  the  one  specified  above  for  the  control  condition. 
Similarities  and  differences  are  highlighted  below: 

The  same  materials  (with  the  exception  of  index  cards  for  updating 
a manual  file)  were  available  for  the  analysts'  use  in  this  condition. 

The  detailed  instructions  tailored  for  this  condition  are  included  as 
Appendix  H.  The  analysts  were  given  the  same  tasks  in  the  same  order 
and  fashion  as  described  for  the  control  subjects.  The  same  basic 
procedure  was  followed  for  questions  1 through  8 as  well  as  for  question 
9:  The  terminal  operator  translated  the  analyst's  request  into  a system 

query,  executed  it,  annotated  the  output,  and  provided  the  annotated 
output  to  the  analyst.  In  the  event  the  information  requested  was  not 
contained  in  the  data  base  (which  was  possible  for  requests  generated  by 
question  9),  a system  query  produced  a negative  reply  and  the  terminal 
operator  informed  the  analyst  of  this,  requesting  that  he  formulate 
another  request.  One  further  comment  should  be  made  regarding  GOQs 
query  elements.  For  each  system  query  generated  in  both  the  control  and 
experimental  conditions,  the  following  elements  were  Included: 

1.  The  coordinates  (location)  of  the  event  or  enemy  unit  identified. 

2.  The  time  the  event  occurred  or  the  unit  was  identified. 


5 Although  the  procedure  used  to  obtain  response  data  for  this  question 
is  described,  these  data  were  not  analyzed.  A breakdown  in  procedure 
for  automatically  recording  (via  a computer  log  tape  file)  the  system 
queries  and  attendant  responses  for  this  question  resulted  in  the 
capturing  of  data  when  anyone  accessed  GOQS  for  system  query  generation. 
Thus,  it  was  not  possible,  in  all  instances,  to  determine  whether  the 
log  tape  entries  were  in  response  to  the  analyst's  requests  during  the 
experiment  or  a result  of  another  user  accessing  the  data  base  through 
GOQS. 
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3.  The  source  that  identified  the  event  or  enemy  unit. 

4.  An  evaluation,  if  available,  of  the  source  that  provided  the 
Information. 

5.  A description  of  the  event  or  activity  reported. 

6.  Any  "remarks"  that  were  provided  by  the  source. 

It  was  necessary  to  include  these  elements  as  part  of  each  system 
query  since  the  pilot  test  had  indicated  that  Intelligence  analysts,  by 
virtue  of  previous  training  and  experience  with  manual  files,  expected 
to  receive  this  information  regardless  of  whether  or  not  they  had 
specifically  requested  it. 


RESULTS 

The  performance  of  analysts  in  both  treatment  conditions  was  evalu- 
ated in  terms  of  three  measures:  timeliness  (task  completion  time), 
accuracy  (the  degree  to  which  the  analyst's  response  correctly  repre- 
sented the  data  that  were  actually  available),  and  completeness  (the 
extent  to  which  the  analyst's  response  Included  all  data  that  were 
available,  regardless  of  the  correctness  of  the  data  actually  reported). 


TIMELINESS 

The  basic  measure  for  timeliness  was  task  completion  time,  the  inter- 
val between  stimulus  (task)  presentation  and  completion  of  the  response. 
For  control  analysts  this  basic  measure  was  entered  into  the  analysis 
just  as  it  was  recorded  for  all  questions  (tasks)  except  numbers  1,  6, 
and  7 where  all  or  part  of  the  data  resided  in  the  computer's  data  base. 
For  these  tasks  in  the  control  condition  and  for  all  tasks  in  the 
experimental  condition,  it  was  necessary  to  correct  the  basic  timeliness 
score  for  output  annotation  and  computer  response  time  (both  post  query 
response  and  "Call  G"  response  intervals).  The  amount  of  time  taken  to 
annotate  the  output  for  a given  question  was  merely  subtracted  from  the 
task  completion  time  score  for  that  question.  Correction  for  computer 
response  was  less  straightforward,  however,  and  was  accomplished  in  the 
following  manner:  Estimates  of  the  two  time  intervals  in  question  were 
obtained  for  each  of  the  eight  tasks  when  the  computer  system  used  in 
the  experiment  was  unburdened6  and  when  few  users  were  accessing  the 
sytem,  under  the  assumption  that  the  unburdened  system's  response  would 
be  similar  to  the  response  that  would  be  obtained  when  using  a dedicated 
system,  as  is  envisioned  for  the  eventual  TOS.  The  difference  between 


6 Subjects  were  run  on  weekdays  between  the  hours  of  0800  to  1700,  an 
interval  when  most  transactions  are  processed  on  the  system  used. 

Thus,  turnaround  was  usually  extremely  variable  and  slow,  in  some  cases 
as  long  as  20  minutes. 


the  basic  timeliness  scores  (corrected  for  output  annotation)  and  the 
unburdened  computer  response  estimates  provided  the  final  corrected 
timeliness  measure.  These  measures  were  summed  across  tasks  to  yield  an 
average  task  completion  time  score  for  each  subject  and  an  analysis  of 
variance  was  performed  on  these  data.  A summary  of  that  analysis  is 
shown  in  Table  2. 


Table  2 

ANOVA  SUMMARY  OF  TASK  COMPLETION  TIME  (IN  SECONDS) 
FOR  EXPERIMENTAL  AND  CONTROL  GROUPS 


Source 

ss 

df 

ms 

f 

P 

Between 

27232.2 

1 

27232.2 

.015 

ns 

Within 

33171322.8 

18 

1842851.27 

Total 

33198555.0 

As  Table  2 shows,  there  was  no  significant  difference  between  the 
experimental  condition  (X  ■ 5387.4,  sd  “ 1776.19)  and  the  control 
condition  (X  ■ 5293.6,  sd  ■ 743.01)  in  terms  of  time  needed  to  complete 
intelligence  tasks  when  corrections  were  made  for  the  potentially 
confounding  variables  mentioned  above.  One  additional  variable,  the 
proficiency  of  the  computer  terminal  operator  (system  query  composition 
time) , was  included  in  the  task  completion  time  score  since  it  could 
have  contaminated  these  results.  The  amount  of  time  needed  to  construct 
system  queries  (including  both  correct  or  incorrect  queries  for  a given 
question)  was  recorded  for  all  system  queries  generated.  The  overlap 
between  conditions  (the  questions  for  which  system  queries  were  generated) 
consisted  of  questions  1,  6,  and  7,  as  indicated  above.  Figures  1,  2, 
and  3 show  query  composition  time  by  trial  separately  for  the  two 
groups,  for  questions  1,  6,  and  7 respectively.  Mean  query  composition 
times  were  calculated  for  the  data  depicted  in  figures  1,  2,  and  3 and 
correlated  t-tests  were  run  to  determine,  by  question,  whether  or  not 
query  composition  time  differed  for  the  two  conditions.  A summary  of 
these  analyses  is  presented  in  Table  3. 


F 


Table  3 

ANALYSES  OF  TERMINAL  OPERATOR  QUERY  COMPOSITION  TIME 
DATA  FOR  EXPERIMENTAL  AND  CONTROL  CONDITIONS 


Experimental  Group 
(n  - 10) 

Control 
(n  = 

Group 

10) 

Question 

X 

sd 

X 

sd 

t 

df 

P 

1 

169.1 

28.96 

183.6 

43.14 

1.70 

9 

ns 

6 

176.0 

29.24 

150.2 

58.62 

1.41 

9 

ns 

7 

100.2 

22.38 

115.6 

38.89 

1.39 

9 

ns 

The  analyses  shown  in  Table  3 clearly  indicate  that  there  were  no 
differences  between  the  performance  of  the  terminal  operator  for  the 
experimental  and  control  conditions.  Although  there  appeard  to  be  some 
improvement  in  the  terminal  operator's  ability  to  construct  system 
queries  as  the  experiment  progressed,  the  experimental  and  control 
trials  were  sequenced  more  or  less  randomly,  which,  apparently,  balanced 
the  operator's  performance  improvement  across  the  two  treatment  condi- 
tions. It  therefore  seems  reasonable  to  conclude  that  operator  profi- 
ciency did  not  systematically  influence  the  timeliness  results  of  the 
experiment. 


ACCURACY 

An  accuracy  score  was  developed  for  each  question  by  assigning  an 
arbitrary  number  of  points  to  that  item.  The  number  of  points  assigned 
for  the  item  was,  in  large  part,  a function  of  the  number  of  elements 
that  a correct  response  would  have  addressed.  Each  element  was  then 
evaluated  for  its  correctness/incorrectness.  If  the  element  was  found 
to  be  incorrect,  one  point  was  subtracted  from  the  total.  Where  the 
subject  plotted  information  (unit  locations,  weapon  sites  etc.),  the 
appropriateness  of  the  symbology  used  and  the  position  of  a plotted  unit 
compared  to  its  actual  or  correct  location  were  considered  to  be  dimen- 
sions of  the  various  elements  of  the  response.  In  regard  to  plotting 
accuracy,  a symbol  was  scored  as  being  accurately  plotted  if  it  fell 
within  a 50  meter  radius  of  the  known  (correct)  location.  A point 
(or  points,  if  the  element  consisted  of  more  than  one  dimension)  was 
subtracted  for  absent  elements,  just  as  if  the  element  had  appeared 
incorrectly.  Thus  an  item  was  not  considered  totally  accurate  if  it 
was  not  complete. 
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QUERY  COMPOSITION  TIME  (IN  SECONDS) 


QUERY  COMPOSITION  TIME  (IN  SECONDS) 


Figure  2.  Terminal  Operator  Query  Composition  Time  for  Task  6 for  Experimental 
and  Control  Groups  across  Trials. 


QUERY  COMPOSITION  TIME  (IN  SECONDS) 


Accuracy  scores  were  summed  across  tasks  and  averaged  to  yield  a mean 
accuracy  score  for  each  participant.  An  analysis  of  variance  was  then 
run  to  determine  whether  the  automated  OB  and  TI  files  significantly 
improved  the  accuracy  of  the  subjects'  performance.  A summary  of  this 
analysis  is  shown  in  Table  4.  This  summary  indicates  that  there  was  a 
highly  significant  difference  between  the  experimental  and  control 
conditions.  Examination  of  mean  scores  (X  = 60.2,  sd  ■ 6.94;  X = 50.2, 
sd  ■ 6.61  for  experimental  and  control  groups,  respectively)  suggests 
that  the  experimental  group's  performance  was  highly  superior  to  that  of 
the  control  group.  Thus,  provided  the  data  base  is  accurate,  it  appears 
that  automation  may  very  well  assist  in  eliminating  many  errors  when 
processing  intelligence  tasks. 


Table  4 

ANOVA  SUMMARY  FOR  ACCURACY  SCORES  OF  SUBJECTS 
IN  EXPERIMENTAL  AND  CONTROL  GROUPS 


Source 

8S 

df 

ms 

f 

P 

Between 

500.0 

1 

500.0 

10.88 

<.001 

Within 

827.2 

18 

45.96 

Total 

1327.2 

COMPLETENESS 

As  was  the  case  with  the  accuracy  criterion,  an  arbitrary  number 
of  points  was  assigned  for  each  question.  The  number  of  points  assigned 
depended  on  the  number  of  elements  that  appeared  in  the  response  (in 
many  cases,  fewer  points  were  assigned  for  completeness  than  for  accuracy 
since  various  dimensions  of  an  element  were  not  considered).  For 
example,  for  those  responses  which  involved  plotting  the  location  of 
enemy  units,  weapon  sites,  etc.,  the  question  was  merely  whether  or  not 
that  unit,  site,  etc.,  had  been  identified,  appropriateness  of  symbology 
and  plotting  accuracy  notwithstanding.  Any  one  given  response,  then, 
was  evaluated  in  terms  of  whether  or  not  each  element  was  present  or 
absent  and  one  point  was  subtracted  from  the  total  for  each  missing 
element.  Completeness  scores  were  averaged  across  the  set  of  eight 
tasks  for  each  participant,  and  an  analysis  of  variance  was  performed  to 
determine  if  the  two  groups  differed  significantly  in  terms  of  this 
criterion.  A summary  of  this  analysis  is  presented  in  Table  5. 


Table  5 


ANOVA  SUMMARY  FOR  COMPLETENESS  SCORES  OF  SUBJECTS 
IN  EXPERIMENTAL  AND  CONTROL  GROUPS 


Source 

ss 

df 

ms 

f 

P 

Between 

140.45 

1 

140.45 

9.46 

<.001 

Within 

267.30 

18 

14.85 

Total 

407.  75 

Table  5 clearly  shows  that  there  was  a significant  difference 
between  the  two  groups.  The  experimental  group's  performance  (X  - 37.4, 
sd  ■ 3.41)  was  significantly  higher  (the  responses  more  complete)  than 
the  control  group's  (X  - 32.1,  sd  - 4.25).  This  finding  was  not  unex- 
pected, given  the  results  obtained  for  the  accuracy  scores  and  the  fact 
that  the  accuracy  and  completeness  criteria  are  correlated  (r  - .99  and 
.93  for  experimental  and  control  groups  respectively). 
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APPENDIX  A REFERENCES  DESCRIBING  THE  CAC  SIMULATED  TOS  AS  IT  EXISTED 
PRIOR  TO  THE  ADDITION  OF  FAP  1 


1.  Reoriented  Tactical  Operations  System,  RTOS/GOQS  Demonstration: 
Revised  RTOS/GOQS  Data  Base  Definitions.  BDM/CARAF  - TR-75-031; 

BDM  Services  Co.,  Combined  Arms  Research  and  Analysis  Facility, 

Fort  Leavenworth,  Kansas,  6 March  1975. 

2.  Reoriented  Tactical  Operations  System,  RTOS/GOQS  Demonstration:  Data 
Element  Dictionary  Current  Formulation  and  Usage.  BDM/CARAF  - 
TR-75-060;  BDM  Services  Co.,  Combined  Arms  Research  and  Analysis 
Facility,  Fort  Leavenworth,  Kansas,  14  May  1975. 

3.  Reoriented  Tactical  Operations  System,  RTOS/GOQS  Demonstration: 

Enemy  Unit  History  File  Requirements  Description.  BDM/CARAF  - 
TR-75-070;  BDM  Services  Co.,  Combined  Arms  Research  and  Analysis 
Facility,  Fort  Leavenworth,  Kansas,  11  June  1975. 

4.  Reoriented  Tactical  Operations  System,  RTOS/GOQS  Demonstration: 

Final  Report.  Volume  III,  Data  Base  Input  User/Planner  Guide. 
BDM/CARAF  - FR-75-074;  BDM  Services  Co.,  Combined  Arms  Research 
and  Analysis  Facility,  Fort  Leavenworth,  Kansas,  7 July  1975. 

5.  Reoriented  Tactical  Operations  System,  RTOS/GOQS  Demonstration: 

Final  Report,  Volume  II.  GOQS  User/Plannner  Guide.  BDM/CARAF  - 
FR-75-075,  BDM  Services  Co.,  Combined  Arms  Research  and  Analysis 
Facility,  Fort  Leavenworth,  Kansas,  7 July  1975. 

6.  Reoriented  Tactical  Operations  System.  RTOS/GOQS  Demonstration: 

Final  Report.  Volume  I:  Executive  Summary.  BDM/CARAF  - FR-75-075; 
BDM  Services  Co.,  Combined  Arms  Research  and  Analysis  Facility, 

Fort  Leavenworth,  Kansas,  7 July  1975. 


APPENDIX  C QUERY  COMPOSITION /ANNOTATION  TIME  FORM  Subject's  Name 


APPENDIX  D PART  I.  PRELIMINARY  INSTRUCTIONS  FOR  SUBJECTS  IN  AUTOMATED 

CONDITION 


You  are  here  today  to  participate  in  an  experiment.  Hie  purpose  of 
the  experiment  is  to  determine  how  well  Intelligence  analysts/OB  techni- 
cians, like  yourself,  can  perform  job  related  tasks  with  either  computer 
data  files  or  manual  and  computer  data  files.  Thus,  some  of  the  partici- 
pants, including  yourself,  will  use  only  the  computer  to  perform  a 
series  of  tasks.  Other  participants  will  have  access  to  some  information 
in  the  computerized  data  files,  but  will  rely  more  heavily  on  manual 
files  to  accomplish  the  same  tasks.  You  will  be  given  more  detailed 
instruction  concerning  what  will  be  required  of  you  later.  On  the 
page  following  these  instructions  is  presented  a detailed  description  of 
the  general  and  special  situation  of  the  54th  Mechanized  Infantry 
Division,  the  unit  to  which  you  will  assume  you  have  been  assigned  for 
the  purposes  of  this  experiment.  The  division  is  part  of  a corps  force 
operating  in  Western  Europe  under  mid-intensity  conflict  conditions. 

Read  these  descriptions  very  carefully  so  that  you  are  fully  read  into 
the  situation.  Also,  you  will  want  to  study  the  map  and  overlay  on  the 
easel  in  front  of  you  which  shows  the  area  in  which  the  battle  is  being 
fought,  the  disposition  of  friendly  units,  and  the  location  of  enemy 
units  that  are  known  to  be  in  contact  on  the  FEBA.  You  will  be  given  15 
minutes  to  study  the  map  and  the  description  of  the  54th  Mechanized 
Infantry  Division.  Do  you  have  any  questions  at  this  point? 
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APPENDIX  D PART  II.  PRELIMINARY  INSTRUCTIONS  FOR  SUBJECTS  IN  MANUAL 

CONDITION 


You  are  here  today  to  participate  in  an  experiment.  The  purpose  of 
the  experiment  is  to  determine  how  well  intelligence  analysts/OB  techni- 
cians, like  yourself,  can  perform  job-related  tasks  with  either  computer 
data  files  or  manual  and  computer  data  files.  Thus,  some  of  the  partici- 
pants will  exclusively  use  the  computer  to  perform  a series  of  tasks. 
Other  participants.  Including  yourself,  will  have  access  to  some  informa- 
tion in  the  computerized  data  file  but  will  rely  more  heavily  on  manual 
files  (OB  and  TI  files)  to  accomplish  the  same  tasks.  You  will  be  given 
more  detailed  instructions  concerning  what  will  be  required  of  you 
later.  On  the  page  following  these  instructions  is  presented  a descrip- 
tion of  the  general  and  special  situation  of  the  54th  Mechanized  Infantry 
Division,  the  unit  to  which  you  will  assume  you  have  been  assigned  for 
the  purposes  of  the  experiment.  The  division  is  part  of  a corps  force 
operating  in  Western  Europe  under  mid-intensity  conflict  conditions. 

Read  these  descriptions  very  carefully  so  that  you  are  fully  read  into 
the  situation.  You  will  want  to  study  the  map  and  overlay  on  the  easel 
in  front  of  you  which  shows  the  area  in  which  the  battle  is  being 
fought,  the  disposition  of  friendly  units,  and  the  location  of  enemy 
units  that  are  known  to  be  in  contact  on  the  FEBA.  You  will  be  given  15 
minutes  to  study  the  description  of  the  54th  Mechanized  Infantry  Division 
and  the  situation  map.  Also,  you  should  become  familiar  with  the 
composition  and  contents  of  the  manual  OB  and  target  intelligence  files 
that  you  will  use  in  the  experiment.  After  you  have  thumbed  through  the 
files,  signal  the  experimenter  and  he  will  give  you  a series  of  four 
questions,  one  at  a time,  for  which  you  will  be  required  to  provide 
answers.  This  should  further  assist  you  in  becoming  familiar  with  and 
extracting  information  from  these  files.  Do  you  have  any  questions  at 
this  point?  If  not,  signal  the  experimenter  when  you  are  finished 
studying  (1)  the  situation  map,  (2)  the  description  of  the  general  and 
special  situation  of  the  54th  Mechanized  Infantry  Division,  and  (3)  your 
manual  files  and  that  you  are  ready  for  the  set  of  questions  that  should 
better  familiarize  you  with  the  manual  files. 
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APPENDIX  E GENERAL  AND  SPECIAL  SITUATIONS  OF  THE  54th  MECHANIZED  INFANTRY 
DIVISION 


1.  General  Situation. 


a.  Warfare  along  the  Western  German  border  was  initiated  by  Parkland 
forces,  an  aggressor  whose  doctrine  and  tactics  are  more  or  less  the 
same  as  the  Soviets',  on  16  August  1975.  Parkland  forces  were  successful 
in  making  gains  all  along  the  Allied  Front  by  using  overpowering  concen- 
tration of  forces  with  powerful  armored  assaults  supported  by  heavy  fire 
power.  Elements  of  the  10th  (US)  Corps  were  able  to  inflict  heavy 
damages  on  attacking  columns  and  following  echelons  through  the  use  of 
tactical  air,  by  using  the  best  defensive  terrain,  and  by  shifting 
forces  within  the  main  battle  area.  The  10th  (US)  Corps  was  successful 
in  conducting  its  planned  retrograde  operation  to  the  Frankfurt-Wiesbaden 
area. 

b.  Friendly  10th  (US)  Corps  is  to  establish  its  defense  on  the 
Frankfurt-Wiesbaden  area  by  221100  August  with  the  23d  Armored  Division, 
54th  Mechanized  Infantry  Division,  and  the  52d  Mechanized  Infantry 
Division  north  to  south  in  that  order.  The  Corps  covering  force  is 
still  in  front  of  the  main  battle  area. 

2.  Special  Situation. 

It  is  now  220600  August.  The  54th  Mechanized  Infantry  Division  has 
just  received  a mission  to  defend  from  MB832806  to  MA913533  NLT  221100 
August;  retain  Frankfurt  north  of  the  Main  River;  establish  covering 
forces  vie  line  blue  NLT  221100  August;  assume  control  of  maneuver 
elements  detached  from  TFNIKF,  vie  line  blue  and  delay  enemy  forward  of 
FEBA  for  a period  of  eight  hours. 
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APPENDIX  F FAMILIARIZATION  TASKS  FOR  CONTROL  GROUP 

Subject  Name 

QUESTION  1 

What  are  the  effective  and  authorized  strengths  of  the  3d  Battalion 
of  the  8th  Field  Artillery  Regiment  of  the  2d  Combined  Arms  Army? 

EM  Officers 

Authorized  strength 
Effective  strength 

QUESTION  2 

Who  is  the  Deputy  Commander  of  the  2d  Combined  Arms  Army? 

Name • Rank_ 

QUESTION  3 

What  Is  the  state  of  training  of  the  20th  Motorized  Rifle  Regiment 
of  the  18th  Mechanized  Infantry  Division? 

Describe  below  the  state  of  training  and  the  regiment's  specialty. 

[ 

QUESTION  4 

How  many  regimental  size  units  have  been  identified  (both  confirmed 
and  unconfirmed)  in  Named  Area  of  Interest 
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APPENDIX  G POST  FAMILIARIZATION  PHASE  INSTRUCTIONS  FOR  CONTROL  GROUP 

For  purposes  of  the  experiment,  you  are  to  assume  that  you  have  just 
been  assigned  to  the  54th  Mechanized  Infantry  Division's  G2  Section  as 
the  operations  sergeant.  The  previous  operations  sergeant  was  killed  a 
very  short  time  ago  when  an  artillery  round  hit  and  destroyed  part  of 
the  G2  Section.  Since  the  shelling,  only  that  portion  of  the  situation 
map  which  you  studied  a short  time  ago  has  been  reconstructed.  However, 
the  manual  OB  and  TI  files  were  salvaged  and  there  are  data  in  some 
computer  files  that  can  be  used  to  reconstruct  the  situation  map  and/or 
to  provide  answers  to  a series  of  nine  questions  that  the  G2  and  the 
commander  have  formulated.  The  experimenter,  playing  the  part  of  the 
G2,  will  ask  you  to  obtain  answers  to  these  questions.  You  will  be 
given  the  questions  one  at  a time  and  you  will  answer  each  question  as 
fully  as  you  can  before  proceeding  to  the  next  question.  You  will  be 
timed,  so  it  is  important  that  you  work  as  quickly,  yet  as  accurately  as 
you  possibly  can.  To  answer  the  questions,  you  will  have,  as  indicated 
above,  two  assets:  your  manual  files  and  the  computer  files.  Mr. 

Elmer,  the  terminal  operator,  will  retrieve  information  for  you  from  the 
computer's  data  base. 

In  order  to  familiarize  you  with  what  data  the  computer  can  provide 
you  and  how  you  should  request  the  information  (through  Mr.  Elmer),  the 
following  instructions  are  provided: 

You  should  think  of  the  computer  as  both  a large  file  cabinet  filled 
with  information  regarding  the  enemy,  and  as  a "superfast"  filing  clerk 
who  can  retrieve  information  for  you  at  a very  rapid  rate. 

In  using  a manual  filing  system,  you  can  only  ask  the  filing  clerk 
to  retrieve  information  for  you.  You  cannot  ask  the  filing  clerk  to 
formulate  an  opinion  or  analyze  the  situation  on  such  matters  as  when, 
where,  or  will  the  enemy  attack.  That  sort  of  analysis  work  is  your 
job.  It  is  the  same  with  the  computer.  It  will  provide  you  information 
quickly,  but  it  will  not  do  the  analysis  for  you.  Here  are  some 
examples  of  the  types  of  questions  that  cannot  be  answered  by  the 
computer. 

1.  When  and  where  will  the  enemy  attack? 

2.  Will  the  enemy  commit  any  reserve  forces? 

3.  What  will  the  commander  of  the  2d  Armored  Division  elect  to  do 
next? 

4.  Based  on  the  trends  of  enemy  unit  movements,  what  should  I do 
next? 
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Here  are  some  examples  of  questions  that  can  be  answered. 

1.  Give  me  a list  of  all  enemy  units  below  battalion  level,  their 
location,  their  authorized  and  effective  strengths  for  personnel  and 
equipment.  The  date  and  time  that  this  information  was  reported,  the 
morale  rating  of  their  units,  and  the  activity  associated  with  them. 

2.  Where  are  all  the  tactical  bridges  located  that  were  reported 
in  the  last  24  hours  in  named  area  of  interest  B?  Also,  specify  the 
type  of  traffic  these  bridges  will  support  and  exactly  when  they  were 
identified . 

3.  Give  me  a list  of  all  subordinate  units  of  the  20th  Motorized 
Rifle  Regiment,  their  location,  what  activity  they  have  been  associated 
with,  when  this  activity  occurred,  what  their  authorized  and  effective 
strengths  are  for  personnel  and  equipment,  what  named  area  of  interest 
they  are  located  in,  and  have  the  output  sorted  by  named  area  of  interest. 

4.  What  reported  enemy  activity  has  occurred  in  the  grid  area  delin- 
eated by  NA0883  in  the  last  10  days?  Tell  me  when  this  activity  occurred, 
where  it  occurred,  what  enemy  unit  was  associated  with  it,  in  what  named 
area  of  interest  it  is  located,  and  sort  the  output  in  chronological 
order . 

You  should  understand  also  that  the  computer  operates  at  a very 
elementary  level.  That  is,  it  provides  information  in  small  bits  (such 
as  the  grid  coordinates  of  an  enemy  unit,  the  time  it  was  sighted,  who 
made  the  sighting,  and  the  reliability  of  the  source).  Therefore,  you 
must  be  certain  to  specify  all  that  you  want  the  computer  to  tell  you 
for  each  question.  For  example,  suppose  you  want  to  know  where  all 
bridges  in  a certain  area  of  the  map  are  located.  You  would  probably 
also  want  to  know  what  type  of  bridge  it  is — tactical  versus  civilian, 
when  it  was  located,  etc.  In  phrasing  your  questions  to  the  operator, 
you  should  specify  all  the  elements  of  information  regarding  the  subject 
you  want  to  know  about.  To  make  it  easier  for  you  to  obtain  all  the 
information  you  might  want,  the  operator  has  been  instructed  to  provide 
the  following  information  for  each  of  the  questions: 

a.  The  coordinates  (location)  of  the  event  or  enemy  unit. 

b.  The  time  the  event  occurred. 

c.  The  source  of  the  information. 

d.  An  evaluation  of  the  source. 

e.  A description  of  the  event  or  unit  identified. 

f.  Any  remarks  that  were  provided  by  the  source  and  entered  into 
the  data  base. 
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If  you  want  more  specifics  than  the  above  elements  will  provide, 
you  must  inform  the  operator.  Do  not  be  concerned  about  requesting  too 
many  specifics,  because  the  operator  will  act  as  an  interpreter  and  will 
screen  out  specifics  which  he  cannot  access  with  the'  system.  He  will 
inform  you  of  instances  such  as  this  or  when  he  cannot  provide  the 
detail  that  you  have  requested. 

The  information  in  the  computer's  data  base  is  cataloged  into  several 
different  files.  These  files  and  a description  of  the  type  of  data 
residing  in  each  is  presented  below. 

ENEMY  UNIT  SITUATION.  This  file  contains  information  taken  from 
raw  data  reports  concerning  an  enemy  unit,  such  as  its  location,  size, 
parent-unit,  the  date-time  that  it  was  located  and/or  the  date-time 
that  the  activity  concerning  this  unit  occurred. 

ENEMY  SITUATION  DATA.  This  file  contains  information  taken  from  raw 
data  reports  concerning  enemy  activity.  It  contains  information  such  as 
reported  enemy  activity  (ambush,  attack,  concentration,  jamming,  etc.), 
the  date-time  that  the  activity  occurred,  where  it  occurred,  the  enemy 
unit  involved,  and  who  reported  the  activity. 

ENEMY  UNIT  HISTORY.  This  file  contains  messages  in  exactly  the  same 
format  as  the  Enemy  Unit  Situation  File.  The  difference  is  that  the 
Enemy  Unit  History  File,  through  its  mode  of  automatic  update  and 
addition,  keeps  a historical  record  of  data  added  to  the  Enemy  Unit 
Situation  File.  This  allows  the  intelligence  analyst  to  analyze  enemy 
unit  movements,  activities,  weapons,  personnel  strengths,  and  trends. 

To  accomplish  the  tasks  that  will  be  given,  you  must  use  data  which 
reside  in  those  files  described  above  and  in  your  manual  OB  and  TI  files. 

The  first  eight  questions  you  will  receive  are  very  elementary 
questions  in  the  sense  that  they  do  not  require  analysis  or  real  correla- 
tion of  data  to  answer.  These  questions  can  be  addressed  to  the  computer 
terminal  operator  more  or  less  as  they  are.  After  you  receive  and  study 
each  of  these  questions  to  determine  what  you  must  provide,  you  will,  in 
turn,  place  a requirement  on  the  termihal  operator.  The  operator  will 
tell  you  where  the  data  you  have  requested  reside — in  the  computer,  your 
manual  files,  or  both.  If  the  information  resides  in  both  the  computer's 
data  base  and  your  manual  files,  the  terminal  operator  will  begin 
retrieving  that  portion  which  is  in  the  computer  data  base.  You  must 
wait  until  he  has  finished  retrieving  this  information  and  provided  it 
to  you  before  you  begin  searching  for  the  rest  of  the  information  in  your 
manual  files.  Of  course,  if  all  the  data  you  have  requested  resides  in 
your  manual  files,  you  will  immediately  begin  searching  for  it  here; 


Question  9,  the  last  one  you  will-  receive,  is  different  from  the 
first  eight  in  that  it  is  complex — it  will  require  your  skills  in  the 
intelligence  area  to  analyze  and  put  together  various  kinds  of  informa- 
tion in  order  to  develop  a conclusion  or  prediction.  The  procedure  that 
you  will  follow  in  obtaining  the  information  you  need  for  this  question 
will  be  slightly  different  from  that  used  for  the  first  group  of  ques- 
tions. Again,  you  will  get  the  question,  study  it  to  determine  what  is 
required  and  the  information  you  will  need  to  satisfy  the  requirement. 
Then,  you  will  ask  Mr.  Elmer  to  provide  the  first  piece  of  information 
you  need.  He  will  translate  your  question  into  a form  which  the  computer 
can  understand  and  attempt  to  obtain  the  information  you  have  requested. 
As  soon  as  he  receives  the  output,  he  will  tell  you  whether  or  not  all 
the  information  available  that  relates  to  your  question  has  come  out  of 
the  computer  or  whether  all  or  part  of  the  information  you  have  requested 
resides  in  your  manual  files.  He  will  also  annotate  the  computer  output 
so  that  you  can  understand  it  and  will  provide  it  to  you.  After  this  is 
done,  and  if  some  of  the  information  that  you  requested  does  reside  in 
your  manual  files,  you  should  immediately  begin  searching  for  it.  If 
the  situation  requires  you  to  search  your  manual  files,  do  not  lay 
another  requirement  on  the  terminal  operator  until  you  finish  retrieving 
all  relevant  information  from  your  manual  files.  You  will  continue  this 
process  until  you  have  obtained  all  the  information  you  feel  you  need  to 
satisfy  the  requirement.  For  this  last  question,  as  well  as  for  the 
previous  ones,  remember  that  you  are  being  timed  and,  thus,  that  you 
» should  work  as  quickly,  but  as  accurately  as  you  can. 


APPENDIX  H POST  FAMILIARIZATION  PHASE  INSTRUCTIONS  FOR  EXPERIMENTAL 
GROUP 


For  purposes  of  the  experiment,  you  are  to  assume  that  you  have  just 
been  assigned  to  the  54th  Mechanized  Infantry  Division's  G2  Section  as 
the  operations  sergeant.  The  previous  operations  sergeant  was  killed  a 
very  short  time  ago  when  an  artillery  round  hit  and  destroyed  part  of 
the  G2  Section.  Since  the  shelling,  only  that  portion  of  the  situation 
map  which  you  studied  a short  time  ago  has  been  reconstructed.  However, 
there  are  data  in  some  computer  files  that  can  be  used  to  reconstruct 
the  situation  map  and/or  to  provide  answers  to  a series  of  nine  questions 
that  the  G2  and  the  commander  have  formulated.  The  experimenter, 
playing  the  part  of  the  G2,  will  ask  you  to  obtain  answers  to  these 
questions.  You  will  be  given  the  questions  one  at  a time  and  you  will 
answer  each  question  as  fully  as  you  can  before  proceeding  to  the  next 
question.  You  will  be  timed,  so  it  is  important  that  you  work  as 
quickly,  yet  as  accurately  as  you  possibly  can.  To  answer  the  questions, 
you  will  have,  as  indicated  above,  the  computer  files.  Mr.  Elmer,  the 
terminal  operator,  will  retrieve  information  for  you  from  the  computer's 
data  base. 

In  order  to  familiarize  you  with  what  data  the  computer  can  provide 
you  and  how  you  should  request  the  information  (through  Mr.  Elmer),  the 
following  instructions  are  provided: 

You  should  think  of  the  computer  as  both  a large  file 
cabinet  filled  with  information  regarding  the  enemy, 
and  as  a "superfast"  filing  clerk  who  can  retrieve 
information  for  you  at  a very  rapid  rate. 

In  using  a manual  filing  system,  you  can  only  ask 
the  filing  clerk  to  retrieve  information  for  you. 

You  cannot  ask  the  filing  clerk  to  formulate  an 
opinion  or  analyze  the  situation  on  such  matters 
as  when,  where,  or  will  the  enemy  attack.  That 
sort  of  analysis  work  is  your  job.  It  is  the  same 
with  the  computer.  It  will  provide  you  information 
quickly,  but  it  will  not  do  the  analysis  for  you. 

Here  are  some  examples  of  the  types  of  questions 
that  cannot  be  answered  by  the  computer. 

1.  When  and  were  will  the  enemy  attack? 

2.  Will  the  enemy  commit  any  reserve  forces? 

3.  What  will  the  commander  of  the  2d  Armored  Division  elect  to  do 
next? 

4.  Based  on  the  trends  of  enemy  unit  movements,  what  should  I do 
next? 


Here  are  some  examples  of  questions  that  can  be  answered. 

1.  Give  me  a list  of  all  enemy  units  below  battalion  level,  their 
location,  their  authorized  and  effective  strengths  for  personnel  and 
equipment.  The  date  and  time  that  this  information  was  reported,  the 
morale  rating  of  their  units,  and  the  activity  associated  with  them. 

2.  Where  are  all  the  tactical  bridges  located  that  were  reported 
in  the  last  24  hours  in  named  area  of  interest  B?  Also,  specify  the 
type  of  traffic  these  bridges  will  support  and  exactly  when  they  were 
identified . 

3.  Give  me  a list  of  all  subordinate  units  of  the  20th  Motorized 
Rifle  Regiment,  their  location,  what  activity  they  have  been  associated 
with,  when  this  activity  occurred,  what  their  authorized  and  effective 
strengths  are  for  personnel  and  equipment,  what  named  area  of  interest 
they  are  located  in,  and  have  the  output  sorted  by  named  area  of  interest. 

4.  What  reported  enemy  activity  has  occurred  in  the  grid  area 
delineated  by  NA0883  in  the  last  10  days?  Tell  me  when  this  activity 
occurred,  where  it  occurred,  what  enemy  unit  was  associated  with  it,  in 
what  named  area  of  interest  it  is  located,  and  sort  the  output  in 
chronological  order. 

You  should  understand  also  that  the  computer  operates  at  a very 
elementary  level.  That  is,  it  provides  information  in  small  bits  (such 
as  the  grid  coordinates  of  an  enemy  unit,  the  time  it  was  sighted,  who 
made  the  sighting,  and  the  reliability  of  the  source).  Therefore,  you 
must  be  certain  to  specify  all  that  you  want  the  computer  to  tell  you 
for  each  question.  For  example,  suppose  you  want  to  know  where  all  the 
bridges  in  a certain  area  of  the  map  are  located.  You  would  probably 
also  want  to  know  what  type  of  bridge  it  is,  tactical  versus  civilian, 
when  it  was  located,  etc.  In  phrasing  your  questions  to  the  operator, 
you  should  specify  all  the  elements  of  information  regarding  the  subject 
that  you  want  to  know  about.  To  make  it  easier  for  you  to  obtain  all 
the  information  you  might  want,  the  operator  has  been  instructed  to 
provide  the  following  Information  for  each  of  the  questions: 

a.  The  coordinates  (location)  of  the  event  or  enemy  unit. 

b.  The  time  the  event  occurred. 

c.  The  source  of  the  Information. 

d.  An  evaluation  of  the  source. 

e.  A description  of  the  event  or  unit  identified. 

f.  Any  remarks  that  were  provided  by  the  source  and  entered  into  the 
data  base. 
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If  you  want  more  specifics  than  the  above  elements  will  provide, 
you  must  inform  the  operator.  Do  not  be  concerned  about  requesting 
too  many  specifics,  because  the  operator  will  act  as  an  interpreter 
and  will  screen  out  specifics  which  he  cannot  access  with  the  system. 

He  will  inform  you  of  instances  such  as  this  or  when  he  cannot  provide 
the  detail  that  you  have  requested. 

The  information  in  the  computer's  data  base  is  cataloged  into 
several  different  files.  These  files  and  a description  of  the  type  of 
data  residing  in  each  is  presented  below. 

ENEMY  UNIT  SITUATION.  - This  file  contains  information  taken  from 
raw  data  reports  concerning  an  enemy  unit  such  as  its  location,  size, 
parent-unit,  the  date-time  that  it  was  located  and/or  the  date-time 
that  the  activity  concerning  this  unit  occurred,  etc. 

ENEMY  SITUATION  DATA.  - This  file  contains  information  taken  from 
raw  data  reports  concerning  enemy  activity.  It  contains  information, 
such  as  reported  enemy  activity  (ambush,  attack,  concentration,  jamming, 
etc.)  the  date-time  that  the  activity  occurred,  where  it  occurred,  the 
enemy  unit  involved,  who  reported  the  activity,  etc. 

ENEMY  UNIT  HISTORY.  - This  file  contains  messages  in  exactly  the 
same  format  as  the  Enemy  Unit  Siutation  File.  The  difference  is  that 
the  Enemy  Unit  History  File,  through  its  mode  of  automatic  update  and 
addition,  keeps  a historical  record  of  data  added  to  the  Enemy  Unit 
Situation  File.  This  allows  the  intelligence  analyst  to  analyze  enemy 
unit  movements,  activities,  weapons,  personnel  strengths,  and  trends. 

ENEMY  ORDER  OF  BATTLE.  - This  file  contains  information  regarding 
enemy  units.  It  contains  such  data  as  parent  units,  subordinate  units, 
unit  commanders,  combat  effectiveness  rating,  equipment  and  personnel 
status  (authorized  vs  on- hand),  equipment,  supply,  morale,  personnel, 
and  training  readiness  ratings  (a  scale  of  1-4  where  1 is  the  highest 
level),  what  weapons  (light,  medium,  and  heavy  tanks,  artillery,  etc.) 
are  organic  to  and  present  in  the  various  units. 

ENEMY  TARGET  INTELLIGENCE.  - This  file  contains  information  regard- 
ing all  known  enemy  targets.  Specific  data  contained  are  target  number, 
a description  of  the  target,  its  altitude,  location,  location  error,  the 
source  of  information,  confirmation  (yes/no),  the  date-time  that  the 
target  was  located,  recommended  means  of  engagement  and  others. 

To  accomplish  the  tasks  you  will  be  given,  you  must  use  data  which 
resides  in  those  files  described  above. 

The  first  eight  questions  you  will  receive  are  very  elementary  ques- 
tions in  the  sense  that  they  do  not  require  analysis  or  real  correlation 
of  data  to  answer.  Each  of  these  questions  can  be  addressed  to  the  com- 
puter terminal  operator  more  or  less  as  they  are.  After  you  receive  and 
study  each  of  these  questions  to  determine  what  you  must  provide,  you  will, 
in  turn,  place  a requirement  on  the  terminal  operator.  The  operator,  in 
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turn,  will  translate  your  request  into  an  appropriate  system  query  or 
queries,  provide  you  the  output,  and  explain  what  the  output  means 
in  terms  you  can  understand . Once  you  get  the  output  and  understand 
what  it  means,  you  will  then  respond  in  the  fashion  specified  for  the 
particular  question  you  are  trying  to  answer. 

Question  9,  the  last  one  you  will  receive,  is  different  from  the 
first  eight  in  that  it  is  complex — it  will  require  your  skills  in 
the  intelligence  area  to  analyze  and  put  together  various  kinds  of 
information  in  order  to  develop  a conclusion  or  prediction.  The  proce- 
dure that  you  will  follow  in  obtaining  the  Information  you  need  for  this 
question  will  be  slightly  different  from  that  used  for  the  first  group 
of  questions.  Again,  you  will  get  the  question,  study  it  to  determine 
what  is  required  and  the  information  you  will  need  to  satisfy  the 
requirement.  Then,  you  will  ask  Mr.  Elmer  to  provide  the  first  piece  of 
information  you  need.  He  will  translate  your  question  into  a form  which 
the  computer  can  understand  and  attempt  to  obtain  the  information  you 
have  requested.  Again,  when  he  receives  the  output,  he  will  provide  it 
to  you  and  translate  it  into  terms  you  can  understand.  If  you  do 
something  with  the  information,  such  as  plot  it  on  the  situation  map, 
you  must  complete  that  activity  before  laying  another  requirement  on  the 
terminal  operator.  You  will  continue  this  process  until  you  have 
obtained  all  the  information  you  feel  you  need  to  satisfy  the  require- 
ment. For  this  last  question,  as  well  as  for  the  previous  ones,  remember 
that  you  are  being  timed  and,  thus,  that  you  should  work  as  quickly,  but 
as  accurately  as  you  can. 
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HQDA  (DAPE  DW) 

HQDA  (DAPE-HRL) 

HQDA  (DAPE  CPS) 

HQDA  (DAFD-MFA) 

HQDA  (DARD-ARS-P) 
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HQOA  (OUSA-OR) 

HQDA  (DAMO-ROR) 

HQDA  (DASG) 

HQDA  (DA10-PI) 

Chitf,  Consult  Div  (DA-OTSG),  Adelphi,  MD 
Mil  Asst.  Hum  Res.  ODDR&E.  OAD  (E&LS) 

HQ  USARAL,  APO  Seattle.  ATTN:  ARAGP-R 

HQ  First  Army,  ATTN:  AFKA-OI-TI 

HQ  Fifth  Army,  Ft  Sem  Houston 

Dir.  Army  St*  Studies  Ofc,  ATTN:  OAVCSA  (DSP) 

Ofc  Chief  of  St*.  Studies  Ofc 

OCSPER,  ATTN:  CPS/OCP 

The  Army  Lib,  Pentagon,  ATTN:  RSB  Chief 

The  Army  Lib.  Pentagon,  ATTN:  ANRAL 

Ofc,  Asst  Sect  of  the  Army  (R&0) 

Tech  Support  Ofc,  OJCS 

USASA,  Arlington,  ATTN:  IARD-T 

USA  Rsch  Ofc,  Durham,  ATTN:  Life  Sciences  Dir 

USARIEM,  Natick,  ATTN:  SGRD-UE-CA 

USATTC,  Ft  Clayton,  ATTN:  STETC-MO-A 

USAIMA,  Ft  Bragg.  ATTN:  ATSU-CTO-OM 

USAIMA,  Ft  Bragg,  ATTN:  Marquat  Lib 

US  WAC  Ctr  & Sch,  Ft  McClellan,  ATTN:  Lib 

US  WAC  Ctr  A Sch,  Ft  McClellan,  ATTN:  Tng  Dir 

USA  Quartermaster  Sch,  Ft  Lee,  ATTN:  ATSM  TE 

Intelligence  Material  Dev  Ofc,  EWL,  Ft  Holabird 

USA  SE  Signal  Sch.  Ft  Gordon,  ATTN:  ATSO-EA 

USA  Chaplain  Ctr  A Sch,  Ft  Hamilton,  ATTN:  ATSC-TE-RD 

USATSCH,  FtEustls,  ATTN:  Educ  Advisor 

USA  War  Collage,  Carlisle  Barracks,  ATTN:  Lib 

WRAIR,  Neuropsychiatry  Div 

DLI,  SDA,  Monterey 

USA  Concept  Anal  Agcv,  Bethetda,  ATTN:  MOCA-WGC 
USA  Concept  Anal  Agcy,  Bathasda,  ATTN:  MOCA-MR 
USA  Concept  Anal  Agcy,  Bethetda,  ATTN:  MOCA-JF 
USA  Artie  Test  Ctr.  APO  Seattle,  ATTN:  STEAC-MO-ASL 
USA  Artie  Test  Ctr,  APO  Seattle,  ATTN:  AMSTE-PL-TS 
USA  Armament  Cmd,  Redstone  Arsenal,  ATTN:  ATSK-TEM 
USA  Armament  Cmd,  Rock  Island,  ATTN:  AMSAR-TDC 


FAANAFEC,  Atlantic  City,  ATTN:  Library 
FAA-NAFEC,  Atlantic  City,  ATTN:  Hum  Engr  Br 
FAA  Aeronautical  Ctr,  Oklahoma  City,  ATTN:  AAC-44D 
USA  Fid  Arty  Sch,  Ft  Sill,  ATTN:  Library 
USA  Armor  Sch,  Ft  Knox,  ATTN:  Library 
USA  Armor  Sch,  Ft  Knox,  ATTN:  ATSB  DI-E 
USA  Armor  Sch,  Ft  Knox,  ATTN:  ATSB-DT  TP 
USA  Armor  Sch,  Ft  Knox,  ATTN:  ATSB-CD-AD 


2 HQUSACDEC,  Ft  Ord,  ATTN:  Library 

1 HQUSACDEC,  Ft  Ord,  ATTN:  ATEC-EX-E-Hum  Factors 

2 USAEEC,  Ft  Benjamin  Harrison,  ATTN:  Library 

1 USAPACDC,  Ft  Benjamin  Harrison,  ATTN:  ATCP-HR 
1 USA  Comm-Elect  Sch,  Ft  Monmouth,  ATTN:  ATSN-EA 
1 USAEC,  Ft  Monmouth,  ATTN:  AMSEL-CT-HDP 
1 USAEC,  Ft  Monmouth,  ATTN:  AMSEL-PA-P 
1 USAEC,  Ft  Monmouth,  ATTN:  AMSEL-SI-CB 
1 USAEC,  Ft  Monmouth,  ATTN:  C,  Fad  Dev  Br 
1 USA  Materials  Sys  Anal  Agcy,  Aberdeen,  ATTN:  AMXSY— P 
1 Edgewood  Arsenal,  Aberdeen,  ATTN:  SAREA-BL-H 

1 USA  Ord  Ctr  & Sch,  Aberdeen,  ATTN:  ATSL-TEM-C 

2 USA  Hum  Engr  Lab,  Aberdeen,  ATTN:  Library /Dir 

1 USA  Combat  Arms  Tng  Bd,  Ft  Banning,  ATTN:  Ad  Supervisor 
1 USA  Infantry  Hum  Rsch  Unit,  Ft  Banning,  ATTN:  Chief 
1 USA  Infantry  Bd,  Ft  Banning,  ATTN:  STEBC— TE— T 
1 USASMA,  Ft  Bliss,  ATTN:  ATSS-LRC 
1 USA  Air  Def  Sch,  Ft  Bliss.  ATTN:  ATSA-CTD— ME 
1 USA  Air  De*  Sch,  Ft  Bliss.  ATTN:  Tech  Lib 
1 USA  Air  Def  Bd,  Ft  Bliss,  ATTN:  FILES 
1 USA  Air  De*  Bd,  Ft  Bliss.  ATTN:  STEBD-PO 
1 USA  Cmd  & General  Stf  College,  Ft  Leavenworth,  ATTN:  Lib 
1 USA  Cmd  & General  Stf  College.  Ft  Leavenworth.  ATTN:  ATSW-SE-L 
1 USA  Cmd  & General  Stf  College,  Ft  Leavenworth,  ATTN:  Ed  Advisor 
1 USA  Combined  Arms  Cmbt  Dev  Act,  Ft  Leavenworth,  ATTN:  DepCdr 
1 USA  Combined  Arms  Cmbt  Dev  Act,  Ft  Leavenworh,  ATTN:  CCS 
1 USA  Combined  Arms  Cmbt  Dev  Act,  Ft  Leavenworth,  ATTN:  ATCASA 
1 USA  Combined  Arms  Cmbt  Dev  Act,  Ft  Leavenworth,  ATTN:  ATCACO— E 
1 USA  Combined  Arms  Cmbt  Dev  Act,  Ft  Leavenworth,  ATTN:  ATCACC— Cl 
1 USAECOM,  Night  Vision  Lab,  Ft  Belvoir,  ATTN:  AMSEL-NV-SD 

3 USA  Computer  Sys  Cmd,  Ft  Belvoir,  ATTN:  Tech  Library 
1 USAMERDC,  Ft  Belvoir,  ATTN:  STSFB-DQ 

1 USA  Eng  Sch.  Ft  Belvoir,  ATTN:  Library 
1 USA  Topographic  Lab,  Ft  Belvoir,  ATTN:  ETL-TD-S 
1 USA  Topographic  Lab,  Ft  Belvoir,  ATTN:  STINFO  Canter 
1 USA  Topographic  Lab,  Ft  Belvoir,  ATTN:  ETL-GSL 
1 USA  Intelligence  Ctr  & Sch,  Ft  Huachuca,  ATTN:  CTD-MS 
1 USA  Intelligence  Ctr  & Sch,  Ft  Huachuca,  ATTN:  ATS— CTD-MS 
1 USA  Intelligence  Ctr  & Sch,  Ft  Huachuca.  ATTN:  ATSI-TE 
1 USA  Intelligence  Ctr  & Sch,  Ft  Huachuca,  ATTN:  ATSI— TEX— GS 
1 USA  Intelligence  Ctr  & Sch,  Ft  Huachuca,  ATTN:  ATSI— CTS-OR 
1 USA  Intelligence  Ctr  & Sch,  Ft  Huachuca.  ATTN:  ATSI— CTO— DT 
1 USA  Intelligence  Ctr  & Sch,  Ft  Huachuca,  ATTN:  ATSI-CTD— CS 
1 USA  Intelligence  Ctr  & Sch,  Ft  Huachuca,  ATTN:  DAS/SRD 
1 USA  Intelligence  Ctr  It  Sch,  Ft  Huachuca,  ATTN:  ATSI-TEM 
1 USA  Intelligence  Ctr  8i  Sch,  Ft  Huachuca,  ATTN:  Library 

1 CDR,  HQ  Ft  Huachuca,  ATTN:  Tech  Re*  Div 

2 CDR,  USA  Electronic  Prvg  Grd,  ATTN:  STEEP-MT-S 
1 CDR.  Project  MASSTER,  ATTN:  Tech  Info  Center 

1 Hq  MASSTER.  USATRADOC,  LNO 
1 Reeearch  Inttltute,  HQ  MASSTER,  Ft  Hood 
1 USA  Recruiting  Cmd,  Ft  Sherdlen,  ATTN:  USARCPM-P 
1 Senior  Army  Adv„  USAFAGOD/TAC,  Elgin  AF  Aux  Fid  No.  9 
1 HQ  USARPAC,  DCSPER,  APO  SF  96668.  ATTN:  GPPE-SE 
1 Stimton  Lib,  Academy  of  Health  Sciencee,  Ft  Sam  Houtton 
1 Marine  Corpt  ln«t.,  ATTN:  Dean -MCI 
1 HOUSMC,  Commandant,  ATTN:  Coda  MTMT  61 

1 HQUSMC,  Commandant,  ATTN:  Coda  MPI -20 

2 USCG  Academy,  New  London,  ATTN:  Admission 
2 USCG  Academy,  New  London,  ATTN:  Library 

1 USCG  Training  Ctr,  NY,  ATTN:  CO 
1 USCG  Training  Ctr,  NY,  ATTN:  Educ  Svc  Ofc 
1 USCG,  Psychol  Res  Br,  DC,  ATTN:  GP  1/82 
1 HQ  Mid-Range  Br,  MC  Det,  Quantico,  ATTN:  P&S  Div 
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1  US  Marine  Corps  Liaision  Otc.  AMC.  Alexandria.  ATTN:  AMCGS-F  1 Def  A Cieil  Inst  of  Enviro  Medicine.  Csntds 
1 USATRADOC.  Ft  Monroe.  ATTN  ATRO  ED  1 AIR  CRESS.  Kenungtor.  ATTN:  Into  Sys  Br 

6 USATRADOC.  Ft  Monroe,  ATTN  ATPR  AD  1 Militaerpeykologisk  Tteneite.  Copehegen 

1 USATRADOC.  Ft  Monroe.  ATTN:  ATTS-EA  ' Military  Attache,  French  EmLnsy,  ATTN  Doc  Sec 

1 USA  Forces  Cmd.  Ft  McPherson,  ATTN:  Librety  ’ Medecm  Chef.  C.E.R.P.A  -Arsenal.  Toulon /Navel  France 

2 USA  Aviation  Test  Bd,  Ft  Rucker,  ATTN:  STEBG-PO  ' Frin  Scientific  Off.  Appl  Hum  Engr  Rsch  Div,  Ministry 

1 USA  Agcv  tor  Aviation  Safety,  Ft  Rucker,  ATTN:  Lihrary  defense.  New  Delhi 

t USA  Agcv  for  Aviation  Safety,  Ft  Rucker.  ATTN:  Educ  Advisor  ' Fers  Fsch  0*c  Library,  AKA,  Israel  Defense  Forces 

1 USA  Aviation  Sch.  Ft  Rucker,  ATTN  PO  Drawer  O 1 Minister*  van  Defensie.  DOOP/KL  Afd  Sociaal 

1 HQUSA  Aviation  Sys  Cmd.  St  Louis,  ATTN:  AMSAV-ZDR  Psychologische  Zaken,  The  Hague.  Netherlands 

2 USA  Aviation  Sys  Test  Act.,  Edwards  AFB,  ATTN:  SAVTE-T 
1 USA  Air  Def  Sch.  Ft  Bliss,  ATTN:  ATSA  TEM 

1 USA  Air  Mobility  Rsch  & Day  Lab,  Moffett  Fid,  ATTN:  SAVDL-AS 
1 USA  Aviation  Sch,  Rat  Tng  Mgt,  Ft  Rucker.  ATTN:  ATST-T-RTM 
1 USA  Aviation  Sch,  CO.  Ft  Rucker,  ATTN:  ATST-D-A 
1 HQ.  USAMC,  Alexandria,  ATTN:  AMXCD-TL 
1 HQ,  USAMC,  Alexandria,  ATTN:  CDR 
1 US  Miliury  Academy,  West  Point,  ATTN:  Serials  Unit 
1 US  Military  Academy,  West  Point,  ATTN:  Ofc  of  Milt  Ldrthp 
1 US  Military  Academy,  West  Point.  ATTN:  MAOR 
t USA  Standardization  Gp,  UK,  FPO  NY,  ATTN:  MASE-GC 
1 Ofc  of  Naval  Rsch,  Arlington,  ATTN:  Code  452 

3 Ofc  of  Naval  Rsch,  Arlington,  ATTN:  Code  458 
1 Ofc  of  Naval  Rsch,  Arlington,  ATTN:  Code  450 

1 Ofc  of  Naval  Rsch,  Arlington,  ATTN:  Code  441  •• 

1 Naval  Aerospc  Med  Res  Lab,  Penucola,  ATTN:  Acout  Sch  Div 
f Naval  Aerospc  Med  Res  Lab,  Pensacola,  ATTN : Code  L51 
1 Naval  Aerospc  Med  Res  Lab.  Pensacola.  ATTN:  Code  L6 
1 Chief  of  NavPers,  ATTN:  Pers-OR 
1 NAVAI RST A.  Norfolk,  ATTN:  Safety  Ctr 
1 Nav  Oceanographic,  DC.  ATTN:  Code  6251,  Charts  5 Tech 
1 Center  of  Naval  Anal,  ATTN:  Doc  Ctr 
1 NavAirSysCom,  ATTN:  AIR-5313C 
1 Nav  BuMed.  ATTN:  713 
1 NavHelicopterSubSqua  2.  FPO  SF  96601 
1 AFHRL  (FT)  William  AFB 
1 AFHRL  ITT)  Lowry  AFB 

1 AFHRL  (ASI  WPAFB,  OH 

2 AFHRL  (DOJZ)  Brooks  AFB 

1 AFHRL  (DOJN)  Lackland  AFB 
1 HQUSAF  (INYSD) 

1 HQUSAF  (DPXXA) 

1 AFVTG  (RD)  Randolph  AFB 

3 AMRL  (HE)  WPAFB,  OH 

2 AF  Inst  of  Tech,  WPAFB,  OH,  ATTN:  ENE/SL 
1 ATC  (XPTDI  Randolph  AFB 

1 USAF  AeroMed  Lib.  Brooks  AFB  (SUL-4),  ATTN:  DOC  SEC 
1 AFOSR  (NL),  Arlington 

1 AF  Log  Cmd,  McClellan  AFB,  ATTN:  ALC/DPCRB 

1 Air  Force  Academy,  CO,  ATTN:  Dept  of  Bel  Sen 
5 NavPers  & Dev  Ctr,  San  Diego 

2 Navy  Med  Neuropsychiatric  Rsch  Unit,  San  Diego 
1 Nav  Electronic  Lab,  San  Diego,  ATTN:  Res  Lab 

1 Nav  TrngCen,  San  Diego,  ATTN:  Code  9000-Lib 
1 NavPostGraSch,  Monterey,  ATTN:  Code  56Ae 
1 NavPostGraSch,  Monterey,  ATTN:  Code  2124 
1 NevTrngEquipCtr,  Orlando,  ATTN:  Tech  Lib 
1 US  Dept  of  Labor,  DC,  ATTN:  Manpower  Admin 
1 US  Dept  of  Justice,  DC,  ATTN:  Drug  Enforce  Admin 
1 Nat  Bur  of  Standards,  DC,  ATTN:  Computer  Info  Section 


1 Net  Clearing  House  for  MH-Info,  Rockville 
1 Denver  Federal  Ctr,  Lakewood,  ATTN:  BLM 
1 2 Defense  Documentation  Center 
4 Dir  Psych,  Army  Hq,  Russell  Ofcs,  Canberra 
1 Scientific  Advsr,  Mil  Bd,  Army  Hq,  Russell  Ofcs,  Canberra 
1 Mil  and  Air  Attache,  Austrian  Embassy 

1 Centre  de  Recherche  Des  Facteurs,  Humaine  de  la  Defense 
Nationale,  Brussels 

2 Canadian  Joint  Sttff  Washington 

1 C/Air  Suff,  Royal  Canadian  AF,  ATTN:  Pars  Std  Anal  Br 

3 Chief,  Canadian  Def  Rsch  Staff,  ATTN:  C/CRDSIW) 

* 4 British  Def  Staff,  British  Embassy,  Washington 


